Visadapt: Catadioptric adaptive camera for scenes of variable density of visual information.

This paper presents the design method of a multi-resolution camera, named Visadapt. It is made of a conventional compact camera with a sensor and a lens pointed to a new deformable mirror so that the mirror in a flat state is parallel to the image plane. The main novelty of the latter mirror, to our knowledge, is the ability to control automatically strokes of several millimeters. This allows Visadapt to capture scenes with a spatially variable density of visual information. A grid of linear actuators, set underneath the mirror surface, deforms the mirror to reach the desired shape computed to capture several areas of different resolutions. Mechanical simulations are allowed to iterate on Visadapt's design, to reduce the geometrical distortions in the images. Evaluations made on an actual prototype of Visadapt show that, by adapting the mirror shape, this camera can magnify a scene object up to 20%, even off-centered in the field-of-view, while still perceiving the rest of the scene.

SofaGym: An Open Platform for Reinforcement Learning Based on Soft Robot Simulations.

OpenAI Gym is one of the standard interfaces used to train Reinforcement Learning (RL) Algorithms. The Simulation Open Framework Architecture (SOFA) is a physics-based engine that is used for soft robotics simulation and control based on real-time models of deformation. The aim of this article is to present SofaGym, an open-source software to create OpenAI Gym interfaces, called environments, out of soft robot digital twins. The link between soft robotics and RL offers new challenges for both fields: representation of the soft robot in an RL context, complex interactions with the environment, use of specific mechanical tools to control soft robots, transfer of policies learned in simulation to the real world, etc. The article presents the large possible uses of SofaGym to tackle these challenges by using RL and planning algorithms. This publication contains neither new algorithms nor new models but proposes a new platform, open to the community, that offers non existing possibilities of coupling RL to physics-based simulation of soft robots. We present 11 environments, representing a wide variety of soft robots and applications; we highlight the challenges showcased by each environment. We propose methods of solving the task using traditional control, RL, and planning and point out research perspectives using the platform.

Visual Haptic Feedback for Training of Robotic Suturing.

Current surgical robotic systems are teleoperated and do not have force feedback. Considerable practice is required to learn how to use visual input such as tissue deformation upon contact as a substitute for tactile sense. Thus, unnecessarily high forces are observed in novices, prior to specific robotic training, and visual force feedback studies demonstrated reduction of applied forces. Simulation exercises with realistic suturing tasks can provide training outside the operating room. This paper presents contributions to realistic interactive suture simulation for training of suturing and knot-tying tasks commonly used in robotically-assisted surgery. To improve the realism of the simulation, we developed a global coordinate wire model with a new constraint development for the elongation. We demonstrated that a continuous modeling of the contacts avoids instabilities during knot tightening. Visual cues are additionally provided, based on the computation of mechanical forces or constraints, to support learning how to dose the forces. The results are integrated into a powerful system-agnostic simulator, and the comparison with equivalent tasks performed with the da Vinci Xi system confirms its realism.

Review on generic methods for mechanical modeling, simulation and control of soft robots.

In this review paper, we are interested in the models and algorithms that allow generic simulation and control of a soft robot. First, we start with a quick overview of modeling approaches for soft robots and available methods for calculating the mechanical compliance, and in particular numerical methods, like real-time Finite Element Method (FEM). We also show how these models can be updated based on sensor data. Then, we are interested in the problem of inverse kinematics, under constraints, with generic solutions without assumption on the robot shape, the type, the placement or the redundancy of the actuators, the material behavior… We are also interested by the use of these models and algorithms in case of contact with the environment. Moreover, we refer to dynamic control algorithms based on mechanical models, allowing for robust control of the positioning of the robot. For each of these aspects, this paper gives a quick overview of the existing methods and a focus on the use of FEM. Finally, we discuss the implementation and our contribution in the field for an open soft robotics research.

A Collapsible Soft Actuator Facilitates Performance in Constrained Environments.

Complex environments, such as those found in surgical and search-and-rescue applications, require soft devices to adapt to minimal space conditions without sacrificing the ability to complete dexterous tasks. Stacked Balloon Actuators (SBAs) are capable of large deformations despite folding nearly flat when deflated, making them ideal candidates for such applications. This paper presents the design, fabrication, modeling, and characterization of monolithic, inflatable, soft SBAs. Modeling is presented using analytical principles based on geometry, and then using conventional and real-time finite element methods. Both one and three degree-of-freedom (DoF) SBAs are fully characterized with regards to stroke, force, and workspace. Finally, three representative demonstrations show the SBA's small-aperture navigation, bracing, and workspace-enhancing capabilities.

Robot-Assisted SpiderMass for In Vivo Real-Time Topography Mass Spectrometry Imaging.

Mass spectrometry imaging (MSI) has shown to bring invaluable information for biological and clinical applications. However, conventional MSI is generally performed ex vivo from tissue sections. Here, we developed a novel MS-based method for in vivo mass spectrometry imaging. By coupling the SpiderMass technology, that provides in vivo minimally invasive analysis-to a robotic arm of high accuracy, we demonstrate that images can be acquired from any surface by moving the laser probe above the surface. By equipping the robotic arm with a sensor, we are also able to both get the topography image of the sample surface and the molecular distribution, and then and plot back the molecular data, directly to the 3D topographical image without the need for image fusion. This is shown for the first time with the 3D topographic MS-based whole-body imaging of a mouse. Enabling fast in vivo MSI bridged to topography paves the way for surgical applications to excision margins.

On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward.

The last five years marked a surge in interest for and use of smart robots, which operate in dynamic and unstructured environments and might interact with humans. We posit that well-validated computer simulation can provide a virtual proving ground that in many cases is instrumental in understanding safely, faster, at lower costs, and more thoroughly how the robots of the future should be designed and controlled for safe operation and improved performance. Against this backdrop, we discuss how simulation can help in robotics, barriers that currently prevent its broad adoption, and potential steps that can eliminate some of these barriers. The points and recommendations made concern the following simulation-in-robotics aspects: simulation of the dynamics of the robot; simulation of the virtual world; simulation of the sensing of this virtual world; simulation of the interaction between the human and the robot; and, in less depth, simulation of the communication between robots. This Perspectives contribution summarizes the points of view that coalesced during a 2018 National Science Foundation/Department of Defense/National Institute for Standards and Technology workshop dedicated to the topic at hand. The meeting brought together participants from a range of organizations, disciplines, and application fields, with expertise at the intersection of robotics, machine learning, and physics-based simulation.

Finite Element Method-Based Kinematics and Closed-Loop Control of Soft, Continuum Manipulators.

This article presents a modeling methodology and experimental validation for soft manipulators to obtain forward kinematic model (FKM) and inverse kinematic model (IKM) under quasi-static conditions (in the literature, these manipulators are usually classified as continuum robots. However, their main characteristic of interest in this article is that they create motion by deformation, as opposed to the classical use of articulations). It offers a way to obtain the kinematic characteristics of this type of soft robots that is suitable for offline path planning and position control. The modeling methodology presented relies on continuum mechanics, which does not provide analytic solutions in the general case. Our approach proposes a real-time numerical integration strategy based on finite element method with a numerical optimization based on Lagrange multipliers to obtain FKM and IKM. To reduce the dimension of the problem, at each step, a projection of the model to the constraint space (gathering actuators, sensors, and end-effector) is performed to obtain the smallest number possible of mathematical equations to be solved. This methodology is applied to obtain the kinematics of two different manipulators with complex structural geometry. An experimental comparison is also performed in one of the robots, between two other geometric approaches and the approach that is showcased in this article. A closed-loop controller based on a state estimator is proposed. The controller is experimentally validated and its robustness is evaluated using Lypunov stability method.

Interactive training system for interventional electrocardiology procedures.

Recent progress in cardiac catheterization and devices has allowed the development of new therapies for severe cardiac diseases like arrhythmias and heart failure. The skills required for such interventions are very challenging to learn, and are typically acquired over several years. Virtual reality simulators may reduce this burden by allowing trainees to practice such procedures without risk to patients. In this paper, we propose the first training system dedicated to cardiac electrophysiology, including pacing and ablation procedures. Our framework involves the simulation of a catheter navigation that reproduces issues intrinsic to intra-cardiac catheterization, and a graphics processing unit (GPU)-based electrophysiological model. A multithreading approach is proposed to compute both physical simulations (navigation and electrophysiology) asynchronously. With this method, we reach computational performances that account for user interactions in real-time. Based on a scenario of cardiac arrhythmia, we demonstrate the ability of the user-guided simulator to navigate inside vessels and cardiac cavities with a catheter and to reproduce an ablation procedure involving: extra-cellular potential measurements, endocardial surface reconstruction, electrophysiology mapping, radio-frequency (RF) ablation, as well as electrical stimulation. A clinical evaluation assessing the different aspects of the simulation is presented. This works is a step towards computerized medical learning curriculum.

Aggregate Constraints for Virtual Manipulation with Soft Fingers.

Interactive dexterous manipulation of virtual objects remains a complex challenge that requires both appropriate hand models and accurate physically-based simulation of interactions. In this paper, we propose an approach based on novel aggregate constraints for simulating dexterous grasping using soft fingers. Our approach aims at improving the computation of contact mechanics when many contact points are involved, by aggregating the multiple contact constraints into a minimal set of constraints. We also introduce a method for non-uniform pressure distribution over the contact surface, to adapt the response when touching sharp edges. We use the Coulomb-Contensou friction model to efficiently simulate tangential and torsional friction. We show through different use cases that our aggregate constraint formulation is well-suited for simulating interactively dexterous manipulation of virtual objects through soft fingers, and efficiently reduces the computation time of constraint solving.

Registration by interactive inverse simulation: application for adaptive radiotherapy.

PURPOSE: This paper introduces a new methodology for semi-automatic registration of anatomical structure deformations. The contribution is to use an interactive inverse simulation of physics-based deformable model, computed in real time. METHODS: The method relies on nonlinear finite element method (FEM) within a constraint-based framework. Given a set of few registered points provided by the user, a real-time optimization adapts the boundary conditions and(/or) some parameters of the FEM in order to obtain the adequate geometrical deformations. To dramatically fasten the process, the method relies on a projection of the model in the space of the optimization variables. In this reduced space, a quadratic programming problem is formulated and solved very quickly. The method is validated with numerical examples for retrieving some unknown parameters such as the Young's modulus and some pressures on the boundaries of the model. RESULTS: The approach is employed in the context of radiotherapy of the neck where weight loss during the 7 weeks of the therapy modifies the volume of the anatomical structures and induces large deformations. Indeed, sensitive structures such as the parotid glands may cross the target volume due to these deformations which leads to adverse effects for the patient. We thus apply the approach for the registration of the parotid glands during the radiotherapy of the head and neck cancer. CONCLUSIONS: The results show how the method could be used in a clinical routine and be employed in the planning in order to limit the radiations of these glands.

Anticipation of brain shift in Deep Brain Stimulation automatic planning.

Deep Brain Stimulation is a neurosurgery procedure consisting in implanting an electrode in a deep structure of the brain. This intervention requires a preoperative planning phase, with a millimetric accuracy, in which surgeons decide the best placement of the electrode depending on a set of surgical rules. However, brain tissues may deform during the surgery because of the brain shift phenomenon, leading the electrode to mistake the target, or moreover to damage a vital anatomical structure. In this paper, we present a patient-specific automatic planning approach for DBS procedures which accounts for brain deformation. Our approach couples an optimization algorithm with FEM based brain shift simulation. The system was tested successfully on a patient-specific 3D model, and was compared to a planning without considering brain shift. The obtained results point out the importance of performing planning in dynamic conditions.

Introducing interactive inverse FEM simulation and its application for adaptive radiotherapy.

We introduce a new methodology for semi-automatic deformable registration of anatomical structures, using interactive inverse simulations. The method relies on non-linear real-time Finite Element Method (FEM) within a constraint-based framework. Given a set of few registered points provided by the user, a real-time optimization adapts the boundary conditions and(/or) some parameters of the FEM in order to obtain the adequate geometrical deformations. To dramatically fasten the process, the method relies on a projection of the model in the space of the optimization variables. In this reduced space, a quadratic programming problem is formulated and solved very quickly. The method is validated with numerical examples for retrieving Young's modulus and some pressures on the boundaries. Then, we apply the approach for the registration of the parotid glands during the radiotherapy of the head and neck cancer. Radiotherapy treatment induces weight loss that modifies the shape and the positions of these structures and they eventually intersect the target volume. We show how we could adapt the planning to limit the radiation of these glands.

Middle-ear microsurgery simulation to improve new robotic procedures.

Otological microsurgery is delicate and requires high dexterity in bad ergonomic conditions. To assist surgeons in these indications, a teleoperated system, called RobOtol, is developed. This robot enhances gesture accuracy and handiness and allows exploration of new procedures for middle ear surgery. To plan new procedures that exploit the capacities given by the robot, a surgical simulator is developed. The simulation reproduces with high fidelity the behavior of the anatomical structures and can also be used as a training tool for an easier control of the robot for surgeons. In the paper, we introduce the middle ear surgical simulation and then we perform virtually two challenging procedures with the robot. We show how interactive simulation can assist in analyzing the benefits of robotics in the case of complex manipulations or ergonomics studies and allow the development of innovative surgical procedures. New robot-based microsurgical procedures are investigated. The improvement offered by RobOtol is also evaluated and discussed.

Computation and visualization of risk assessment in deep brain stimulation planning.

Deep Brain Stimulation is aneurosurgical approach for the treatment of pathologies such as Parkinson's disease. The basic principle consists in placing a thin electrode in a deep part of the brain. To safely reach the target of interest, careful planning must be performed to ensure that no vital structure (e.g. blood vessel) will be damaged during the insertion of the electrode. Currently this planning phase is done without considering the brain shift, which occurs during the surgery once the skull is open, leading to increased risks of complications. In this paper, we propose a method to compute the motion of anatomical structures induced by the brain shift. This computation is based on a biomechanical model of the brain and the cerebro-spinal fluid. We then visualize in a intuitive way the risk of damaging vital structures with the electrode.

Constraint-based simulation for non-rigid real-time registration.

In this paper we propose a method to address the problem of non-rigid registration in real-time. We use Lagrange multipliers and soft sliding constraints to combine data acquired from dynamic image sequence and a biomechanical model of the structure of interest. The biomechanical model plays a role of regularization to improve the robustness and the flexibility of the registration. We apply our method to a pre-operative 3D CT scan of a porcine liver that is registered to a sequence of 2D dynamic MRI slices during the respiratory motion. The finite element simulation provides a full 3D representation (including heterogeneities such as vessels, tumor,...) of the anatomical structure in real-time.

Complete Real-Time Liver Model Including Glisson's Capsule, Vascularization and Parenchyma.

Accurate biomechanical modeling of liver is of paramount interest in pre-operative planning or computer-aided per-operative guidance. Since the liver is an organ composed of three different components (parenchyma, vascularization and Glisson's capsule), an efficient and realistic simulation of its behaviour is a challenging task. In this paper we propose a complete model of liver where each component is modelled with different type of finite elements chosen according to the nature and mechanical properties of the component. The elements of different types are coupled via mechanical mapping encoded in the global stiffness matrix. In the result section, we first focus on simulation of Glisson's capsule using constant-strain triangular elements: we compare the model to a detailed non-real-time model and also reproduce previously published aspiration test showing the importance of the capsule. Finally, we demonstrate that the proposed complete liver model can be used in a real-time simulation.

Real-time simulation of contact and cutting of heterogeneous soft-tissues.

This paper presents a numerical method for interactive (real-time) simulations, which considerably improves the accuracy of the response of heterogeneous soft-tissue models undergoing contact, cutting and other topological changes. We provide an integrated methodology able to deal both with the ill-conditioning issues associated with material heterogeneities, contact boundary conditions which are one of the main sources of inaccuracies, and cutting which is one of the most challenging issues in interactive simulations. Our approach is based on an implicit time integration of a non-linear finite element model. To enable real-time computations, we propose a new preconditioning technique, based on an asynchronous update at low frequency. The preconditioner is not only used to improve the computation of the deformation of the tissues, but also to simulate the contact response of homogeneous and heterogeneous bodies with the same accuracy. We also address the problem of cutting the heterogeneous structures and propose a method to update the preconditioner according to the topological modifications. Finally, we apply our approach to three challenging demonstrators: (i) a simulation of cataract surgery (ii) a simulation of laparoscopic hepatectomy (iii) a brain tumor surgery.

Validation method of a middle ear mechanical model to develop a surgical simulator.

Ossicular surgery requires a high dexterity for the manipulation of the fragile and small middle ear components. Currently, the only efficient technique for training residents in otological surgery is through the use of temporal bone specimens, where any existing surgical simulator does not provide useful feedback. The objective of this study was to develop a finite-element model of the human ossicular chain dedicated to surgical simulation and to propose a method to evaluate its behavior. A model was developed based on human middle ear micromagnetic resonance imaging. The mechanical parameters were determined according to published data. To assess its performance, the middle ear transfer function was analyzed. The robustness of our model and the influence of different middle ear components were also evaluated at low frequency by static force pressure simulations. The mechanical behavior of our model in nominal and pathological conditions was in good agreement with published human temporal bone measurements. We showed that the cochlea influences the transfer function only at high frequency and could be omitted from a surgical simulator. In addition, surgeons were able to manipulate the validated middle ear model with a real-time haptic feedback. The computational efficiency of our approach allowed real-time interactions, making it suitable for use in a training simulator.

Towards an interactive electromechanical model of the heart.

In this work, we develop an interactive framework for rehearsal of and training in cardiac catheter ablation, and for planning cardiac resynchronization therapy. To this end, an interactive and real-time electrophysiology model of the heart is developed to fit patient-specific data. The proposed interactive framework relies on two main contributions. First, an efficient implementation of cardiac electrophysiology is proposed, using the latest graphics processing unit computing techniques. Second, a mechanical simulation is then coupled to the electrophysiological signals to produce realistic motion of the heart. We demonstrate that pathological mechanical and electrophysiological behaviour can be simulated.